{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the second project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import json\n",
    "\n",
    "import numpy as np\n",
    "import random\n",
    "import copy\n",
    "from collections import namedtuple, deque\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Reacher.app\"`\n",
    "- **Windows** (x86): `\"path/to/Reacher_Windows_x86/Reacher.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Reacher_Windows_x86_64/Reacher.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Reacher_Linux/Reacher.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Reacher_Linux/Reacher.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Reacher.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Reacher.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_size -> 5.0\n",
      "\t\tgoal_speed -> 1.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "# For local machine\n",
    "from unityagents import UnityEnvironment\n",
    "# env = UnityEnvironment(file_name='./Reacher_Linux/Reacher.x86_64')\n",
    "env = UnityEnvironment(file_name='./Reacher_Linux_20/Reacher.x86_64')\n",
    "\n",
    "# # For Udacity Workspace\n",
    "# !pip -q install ./python\n",
    "# from unityagents import UnityEnvironment\n",
    "\n",
    "# # # select this option to load version 1 (with a single agent) of the environment\n",
    "# # env = UnityEnvironment(file_name='/data/Reacher_One_Linux_NoVis/Reacher_One_Linux_NoVis.x86_64')\n",
    "\n",
    "# # select this option to load version 2 (with 20 agents) of the environment\n",
    "# env = UnityEnvironment(file_name='/data/Reacher_Linux_NoVis/Reacher.x86_64')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, a double-jointed arm can move to target locations. A reward of `+0.1` is provided for each step that the agent's hand is in the goal location. Thus, the goal of your agent is to maintain its position at the target location for as many time steps as possible.\n",
    "\n",
    "The observation space consists of `33` variables corresponding to position, rotation, velocity, and angular velocities of the arm.  Each action is a vector with four numbers, corresponding to torque applicable to two joints.  Every entry in the action vector must be a number between `-1` and `1`.\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 1 1\n",
      "Size of each action: 4\n",
      "There are 1 agents. Each observes a state with length: 33\n",
      "The state for the first agent looks like: [ 0.00000000e+00 -4.00000000e+00  0.00000000e+00  1.00000000e+00\n",
      " -0.00000000e+00 -0.00000000e+00 -4.37113883e-08  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.00000000e+01  0.00000000e+00\n",
      "  1.00000000e+00 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  5.75471878e+00 -1.00000000e+00\n",
      "  5.55726671e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00\n",
      " -1.68164849e-01]\n",
      "(33,) (33,)\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents, env_info.vector_observations.shape[0])\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])\n",
    "\n",
    "print(states[0].shape, states[0,:].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Agent Training (Using DDPG)\n",
    "\n",
    "Train the agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Actor and Critic Networks\n",
    "def hidden_init(layer):\n",
    "    fan_in = layer.weight.data.size()[0]\n",
    "    lim = 1. / np.sqrt(fan_in)\n",
    "    return (-lim, lim)\n",
    "\n",
    "\n",
    "class Actor(nn.Module):\n",
    "    \"\"\"Actor (Policy) Model.\"\"\"\n",
    "\n",
    "    def __init__(self, state_size, action_size, seed, fc1_units=256, fc2_units=128):\n",
    "        \"\"\"Initialize parameters and build model.\n",
    "        Params\n",
    "        ======\n",
    "            state_size (int): Dimension of each state\n",
    "            action_size (int): Dimension of each action\n",
    "            seed (int): Random seed\n",
    "            fc1_units (int): Number of nodes in first hidden layer\n",
    "            fc2_units (int): Number of nodes in second hidden layer\n",
    "        \"\"\"\n",
    "        super(Actor, self).__init__()\n",
    "        self.seed = torch.manual_seed(seed)\n",
    "        self.fc1 = nn.Linear(state_size, fc1_units)\n",
    "        self.fc2 = nn.Linear(fc1_units, fc2_units)\n",
    "        self.fc3 = nn.Linear(fc2_units, action_size)\n",
    "        self.reset_parameters()\n",
    "\n",
    "    def reset_parameters(self):\n",
    "        self.fc1.weight.data.uniform_(*hidden_init(self.fc1))\n",
    "        self.fc2.weight.data.uniform_(*hidden_init(self.fc2))\n",
    "        self.fc3.weight.data.uniform_(-3e-3, 3e-3)\n",
    "\n",
    "    def forward(self, state):\n",
    "        \"\"\"Build an actor (policy) network that maps states -> actions.\"\"\"\n",
    "        x = F.elu(self.fc1(state))\n",
    "        x = F.elu(self.fc2(x))\n",
    "        return F.tanh(self.fc3(x))\n",
    "\n",
    "\n",
    "class Critic(nn.Module):\n",
    "    \"\"\"Critic (Value) Model.\"\"\"\n",
    "\n",
    "    def __init__(self, state_size, action_size, seed, fcs1_units=256, fc2_units=128):\n",
    "        \"\"\"Initialize parameters and build model.\n",
    "        Params\n",
    "        ======\n",
    "            state_size (int): Dimension of each state\n",
    "            action_size (int): Dimension of each action\n",
    "            seed (int): Random seed\n",
    "            fcs1_units (int): Number of nodes in the first hidden layer\n",
    "            fc2_units (int): Number of nodes in the second hidden layer\n",
    "        \"\"\"\n",
    "        super(Critic, self).__init__()\n",
    "        self.seed = torch.manual_seed(seed)\n",
    "        self.fcs1 = nn.Linear(state_size, fcs1_units)\n",
    "        self.fc2 = nn.Linear(fcs1_units+action_size, fc2_units)\n",
    "        self.fc3 = nn.Linear(fc2_units, 1)\n",
    "        self.reset_parameters()\n",
    "\n",
    "    def reset_parameters(self):\n",
    "        self.fcs1.weight.data.uniform_(*hidden_init(self.fcs1))\n",
    "        self.fc2.weight.data.uniform_(*hidden_init(self.fc2))\n",
    "        self.fc3.weight.data.uniform_(-3e-3, 3e-3)\n",
    "\n",
    "    def forward(self, state, action):\n",
    "        \"\"\"Build a critic (value) network that maps (state, action) pairs -> Q-values.\"\"\"\n",
    "        xs = F.elu(self.fcs1(state))\n",
    "        x = torch.cat((xs, action), dim=1)\n",
    "        x = F.elu(self.fc2(x))\n",
    "        return self.fc3(x)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Agent\n",
    "BUFFER_SIZE = int(1e5)  # replay buffer size\n",
    "BATCH_SIZE = 128        # minibatch size\n",
    "GAMMA = 0.99            # discount factor\n",
    "TAU = 1e-3              # for soft update of target parameters\n",
    "LR_ACTOR = 1e-4 #3e-5 #1e-4         # learning rate of the actor \n",
    "LR_CRITIC = 1e-4 #3e-5 #1e-4        # learning rate of the critic\n",
    "WEIGHT_DECAY_actor = 0.0 #3e-4 #0        # L2 weight decay\n",
    "WEIGHT_DECAY_critic = 0.0 #1e-6 #0        # L2 weight decay\n",
    "\n",
    "#to decay exploration as it learns\n",
    "EPS_START=1.0\n",
    "EPS_END=0.05\n",
    "EPS_DECAY=3e-5\n",
    "\n",
    "class Agent():\n",
    "    \"\"\"Interacts with and learns from the environment.\"\"\"\n",
    "    \n",
    "    def __init__(self, state_size, action_size, num_agents, random_seed):\n",
    "        \"\"\"Initialize an Agent object.\n",
    "        \n",
    "        Params\n",
    "        ======\n",
    "            state_size (int): dimension of each state\n",
    "            action_size (int): dimension of each action\n",
    "            random_seed (int): random seed\n",
    "        \"\"\"\n",
    "        self.state_size = state_size\n",
    "        self.action_size = action_size\n",
    "        self.num_agents = num_agents\n",
    "        self.seed = random.seed(random_seed)\n",
    "\n",
    "        # Actor Network (w/ Target Network)\n",
    "        self.actor_local = Actor(state_size, action_size, random_seed).to(device)\n",
    "        self.actor_target = Actor(state_size, action_size, random_seed).to(device)\n",
    "        self.actor_optimizer = optim.Adam(self.actor_local.parameters(), lr=LR_ACTOR, weight_decay=WEIGHT_DECAY_actor)\n",
    "\n",
    "        # Critic Network (w/ Target Network)\n",
    "        self.critic_local = Critic(state_size, action_size, random_seed).to(device)\n",
    "        self.critic_target = Critic(state_size, action_size, random_seed).to(device)\n",
    "        self.critic_optimizer = optim.Adam(self.critic_local.parameters(), lr=LR_CRITIC, weight_decay=WEIGHT_DECAY_critic)\n",
    "\n",
    "        # Noise process\n",
    "        #self.noise = OUNoise(action_size, random_seed) #single agent only\n",
    "        self.noise = OUNoise((num_agents, action_size), random_seed) #both singe and multiple agent\n",
    "        self.eps = EPS_START\n",
    "\n",
    "        # Replay memory\n",
    "        self.memory = ReplayBuffer(action_size, BUFFER_SIZE, BATCH_SIZE, random_seed)\n",
    "    \n",
    "        # Make sure target is initialized with the same weight as the source (found on slack to make big difference)\n",
    "        self.hard_update(self.actor_target, self.actor_local)\n",
    "        self.hard_update(self.critic_target, self.critic_local)\n",
    "\n",
    "\n",
    "    def step(self, states, actions, rewards, next_states, dones):\n",
    "        \"\"\"Save experience in replay memory, and use random sample from buffer to learn.\"\"\"\n",
    "        # Save experience / reward\n",
    "        #Experience from each agent is separately saved (so it works for single or multi agent environment)\n",
    "        #This works because each agent is operating in a separate/independent environment\n",
    "        for a in range(self.num_agents):\n",
    "            self.memory.add(states[a], actions[a], rewards[a], next_states[a], dones[a])\n",
    "\n",
    "        # Learn, if enough samples are available in memory\n",
    "        if len(self.memory) > BATCH_SIZE:\n",
    "            experiences = self.memory.sample()\n",
    "            self.learn(experiences, GAMMA)\n",
    "\n",
    "    def act(self, states, add_noise=True):\n",
    "        \"\"\"Returns actions for given state as per current policy.\"\"\"\n",
    "        states = torch.from_numpy(states).float().to(device)\n",
    "        self.actor_local.eval()\n",
    "        with torch.no_grad():\n",
    "            actions = self.actor_local(states).cpu().data.numpy()\n",
    "        self.actor_local.train()\n",
    "        \n",
    "        #add noise according to epsilon probability\n",
    "        if add_noise and (np.random.random() < self.eps):\n",
    "            actions += self.noise.sample()\n",
    "            #update the exploration parameter\n",
    "            self.eps -= EPS_DECAY\n",
    "            if self.eps < EPS_END:\n",
    "                self.eps = EPS_END\n",
    "            #self.noise.reset() #not sure if need to do this here\n",
    "\n",
    "        return np.clip(actions, -1, 1)\n",
    "\n",
    "    def reset(self):\n",
    "        self.noise.reset()\n",
    "\n",
    "    def learn(self, experiences, gamma):\n",
    "        \"\"\"Update policy and value parameters using given batch of experience tuples.\n",
    "        Q_targets = r + γ * critic_target(next_state, actor_target(next_state))\n",
    "        where:\n",
    "            actor_target(state) -> action\n",
    "            critic_target(state, action) -> Q-value\n",
    "\n",
    "        Params\n",
    "        ======\n",
    "            experiences (Tuple[torch.Tensor]): tuple of (s, a, r, s', done) tuples \n",
    "            gamma (float): discount factor\n",
    "        \"\"\"\n",
    "        states, actions, rewards, next_states, dones = experiences\n",
    "\n",
    "        # ---------------------------- update critic ---------------------------- #\n",
    "        # Get predicted next-state actions and Q values from target models\n",
    "        actions_next = self.actor_target(next_states)\n",
    "        Q_targets_next = self.critic_target(next_states, actions_next)\n",
    "        # Compute Q targets for current states (y_i)\n",
    "        Q_targets = rewards + (gamma * Q_targets_next * (1 - dones))\n",
    "        # Compute critic loss\n",
    "        Q_expected = self.critic_local(states, actions)\n",
    "        critic_loss = F.mse_loss(Q_expected, Q_targets)\n",
    "        # Minimize the loss\n",
    "        self.critic_optimizer.zero_grad()\n",
    "        critic_loss.backward()\n",
    "        torch.nn.utils.clip_grad_norm(self.critic_local.parameters(), 1.0) #clip the gradient for the critic network (Udacity hint)\n",
    "        self.critic_optimizer.step()\n",
    "\n",
    "        # ---------------------------- update actor ---------------------------- #\n",
    "        # Compute actor loss\n",
    "        actions_pred = self.actor_local(states)\n",
    "        actor_loss = -self.critic_local(states, actions_pred).mean()\n",
    "        # Minimize the loss\n",
    "        self.actor_optimizer.zero_grad()\n",
    "        actor_loss.backward()\n",
    "        self.actor_optimizer.step()\n",
    "\n",
    "        # ----------------------- update target networks ----------------------- #\n",
    "        self.soft_update(self.critic_local, self.critic_target, TAU)\n",
    "        self.soft_update(self.actor_local, self.actor_target, TAU)                     \n",
    "\n",
    "    def soft_update(self, local_model, target_model, tau):\n",
    "        \"\"\"Soft update model parameters.\n",
    "        θ_target = τ*θ_local + (1 - τ)*θ_target\n",
    "\n",
    "        Params\n",
    "        ======\n",
    "            local_model: PyTorch model (weights will be copied from)\n",
    "            target_model: PyTorch model (weights will be copied to)\n",
    "            tau (float): interpolation parameter \n",
    "        \"\"\"\n",
    "        for target_param, local_param in zip(target_model.parameters(), local_model.parameters()):\n",
    "            target_param.data.copy_(tau*local_param.data + (1.0-tau)*target_param.data)\n",
    "\n",
    "\n",
    "    ## from slack - Since you're using DDPG, @gregoriomezquita mentioned that \n",
    "    ## initializing the weights of the target networks to be the same as those \n",
    "    ## of the live networks seemed to make a huge difference\n",
    "    def hard_update(self, target, source):\n",
    "        for target_param, source_param in zip(target.parameters(), source.parameters()):\n",
    "            target_param.data.copy_(source_param.data)\n",
    "\n",
    "\n",
    "class OUNoise:\n",
    "    \"\"\"Ornstein-Uhlenbeck process.\"\"\"\n",
    "\n",
    "    def __init__(self, size, seed, mu=0., theta=0.15, sigma=0.2):\n",
    "        \"\"\"Initialize parameters and noise process.\"\"\"\n",
    "        self.mu = mu * np.ones(size)\n",
    "        self.theta = theta\n",
    "        self.sigma = sigma\n",
    "        self.seed = random.seed(seed)\n",
    "        self.size = size\n",
    "        self.reset()\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\"Reset the internal state (= noise) to mean (mu).\"\"\"\n",
    "        self.state = copy.copy(self.mu)\n",
    "\n",
    "    def sample(self):\n",
    "        \"\"\"Update internal state and return it as a noise sample.\"\"\"\n",
    "        x = self.state\n",
    "        #dx = self.theta * (self.mu - x) + self.sigma * np.array([random.random() for i in range(len(x))])\n",
    "        dx = self.theta * (self.mu - x) + self.sigma * np.random.standard_normal(self.size)\n",
    "        self.state = x + dx\n",
    "        return self.state\n",
    "\n",
    "class ReplayBuffer:\n",
    "    \"\"\"Fixed-size buffer to store experience tuples.\"\"\"\n",
    "\n",
    "    def __init__(self, action_size, buffer_size, batch_size, seed):\n",
    "        \"\"\"Initialize a ReplayBuffer object.\n",
    "        Params\n",
    "        ======\n",
    "            buffer_size (int): maximum size of buffer\n",
    "            batch_size (int): size of each training batch\n",
    "        \"\"\"\n",
    "        self.action_size = action_size\n",
    "        self.memory = deque(maxlen=buffer_size)  # internal memory (deque)\n",
    "        self.batch_size = batch_size\n",
    "        self.experience = namedtuple(\"Experience\", field_names=[\"state\", \"action\", \"reward\", \"next_state\", \"done\"])\n",
    "        self.seed = random.seed(seed)\n",
    "    \n",
    "    def add(self, state, action, reward, next_state, done):\n",
    "        \"\"\"Add a new experience to memory.\"\"\"\n",
    "        e = self.experience(state, action, reward, next_state, done)\n",
    "        self.memory.append(e)\n",
    "    \n",
    "    def sample(self):\n",
    "        \"\"\"Randomly sample a batch of experiences from memory.\"\"\"\n",
    "        experiences = random.sample(self.memory, k=self.batch_size)\n",
    "\n",
    "        states = torch.from_numpy(np.vstack([e.state for e in experiences if e is not None])).float().to(device)\n",
    "        actions = torch.from_numpy(np.vstack([e.action for e in experiences if e is not None])).float().to(device)\n",
    "        rewards = torch.from_numpy(np.vstack([e.reward for e in experiences if e is not None])).float().to(device)\n",
    "        next_states = torch.from_numpy(np.vstack([e.next_state for e in experiences if e is not None])).float().to(device)\n",
    "        dones = torch.from_numpy(np.vstack([e.done for e in experiences if e is not None]).astype(np.uint8)).float().to(device)\n",
    "\n",
    "        return (states, actions, rewards, next_states, dones)\n",
    "\n",
    "    def __len__(self):\n",
    "        \"\"\"Return the current size of internal memory.\"\"\"\n",
    "        return len(self.memory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "agent = Agent(state_size=env_info.vector_observations.shape[1], action_size=brain.vector_action_space_size, \n",
    "              num_agents=env_info.vector_observations.shape[0],  random_seed=0)\n",
    "\n",
    "\n",
    "#for single and multiple agents\n",
    "def ddpg(n_episodes=2000):\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    scores_list = []\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]     # reset the environment\n",
    "        states = env_info.vector_observations                   # get the current states (for all agents)\n",
    "        agent.reset() #reset the agent OU Noise\n",
    "        scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "        while True:\n",
    "            actions = agent.act(states)\n",
    "            env_info = env.step(actions)[brain_name]           # send all actions to the environment\n",
    "            next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "            rewards = env_info.rewards                         # get rewards (for each agent)\n",
    "            dones = env_info.local_done                        # see if episode finished\n",
    "            scores += rewards                                  # update the score (for each agent)\n",
    "            agent.step(states, actions, rewards, next_states, dones) #train the agent\n",
    "            \n",
    "            # Extra Learning per time step (since generating so much experience at each step)\n",
    "            if len(agent.memory) > BATCH_SIZE:\n",
    "                for _ in range(3):\n",
    "                    experiences = agent.memory.sample()\n",
    "                    agent.learn(experiences, GAMMA)\n",
    "            \n",
    "            states = next_states                               # roll over states to next time step\n",
    "            if np.any(dones):                                  # exit loop if episode finished\n",
    "                break\n",
    "            #print('Total score (averaged over agents) this episode: {}'.format(np.mean(score)))\n",
    "        \n",
    "        scores_deque.append(np.mean(scores))\n",
    "        scores_list.append(np.mean(scores))\n",
    "        \n",
    "        #print('\\rEpisode {}\\tAverage Score: {:.2f}\\tScore: {}'.format(i_episode, np.mean(scores_deque), score), end=\"\")\n",
    "        print('Episode {}\\tAverage Score: {:.2f}\\tScore: {}'.format(i_episode, np.mean(scores_deque), np.mean(scores)))\n",
    "        print('Epsilon: {} and Memory size: {}'.format(agent.eps, len(agent.memory)))\n",
    "        \n",
    "        if i_episode % 100 == 0:\n",
    "            torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))\n",
    "            \n",
    "        if np.mean(scores_deque) > 30 and len(scores_deque) >= 100:\n",
    "            torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))\n",
    "            break\n",
    "            \n",
    "    return scores_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Use actiove_session() on Udacity workspace\n",
    "from workspace_utils import active_session\n",
    "\n",
    "with active_session():\n",
    "    # do long-running work here\n",
    "    scores = ddpg()\n",
    "    with open(\"scores_file.json\", \"w\") as write_file:\n",
    "        json.dump(scores, write_file)\n",
    "\n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(111)\n",
    "    plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "    plt.ylabel('Score')\n",
    "    plt.xlabel('Episode #')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8VfX9+PHXO3svEpJA2DtsiKBFFFEU96gVV7VWS61t7Xa0/bZ2/arfWldtrVvbusfXVSeIioMRdtibJITsHTLv+/fHvQkBEnIzbm5y7/v5eOTBveeek/M+HDjv+9miqhhjjPFfAd4OwBhjjHdZIjDGGD9nicAYY/ycJQJjjPFzlgiMMcbPWSIwxhg/Z4nAGGP8nCUCY4zxc5YIjDHGzwV5OwB3JCYm6vDhw70dhjHG9Ctr1qwpUtWkjvbrF4lg+PDhZGZmejsMY4zpV0Rkvzv7WdWQMcb4OY8nAhEJFJF1IvKO6/0IEVkpIrtE5CURCfF0DMYYY9rXGyWCHwFbW72/B7hfVUcDpcCNvRCDMcaYdng0EYhIGnA+8ITrvQDzgVdduzwLXOLJGIwxxpyYp0sEDwC3AQ7X+wFAmao2ut7nAIM9HIMxxpgT8FgiEJELgAJVXdPF4xeLSKaIZBYWFvZwdMYYY5p5skQwB7hIRPYBL+KsEnoQiBOR5m6raUBuWwer6mOqmqGqGUlJHXaDNcYY00UeSwSqeqeqpqnqcOBK4GNVvQZYBlzu2u164E1PxWA6p7HJwbLtBbyfdYjahqaW7fWNDj7cfIhHPtnN2xsOsiG7jKq6xuOOV1UKKmrb/N2FlXU0OWxZVGP6Im8MKLsdeFFE/gisA570Qgymldyyw7y46gCvZOZwyPUgjwoN4pyJKUSHBfHWhoOUVNcfdUxUaBAPLJrGWenJANQ2NPHzVzbwzsY8fjh/ND85aywBAUJjk4M/vbuVp7/YR0pMGJdMH8xlMwYzNjm616/T0+obHWzJqyAoQJiQGkNggHg7JGPcIv1h8fqMjAy1kcU9L7+iloc/3sWLqw/Q6FBOH5vElScNJSo0iDfX5/J+1iHqGh0sSE/m6zMHM3NYAnnlh9lfXMPDH+8i62A5ty8cz+Uz01j8r0zWHihj1ogEVu0t4awJydx1UTp3vLaJz3cVcfnMNEqq6/l0RyFNDuX7Z4zi52ePw9mR7HjFVXVEhAQRHhLYrWvMKa0hK7echZNSu/V7DpXXcuuL67j25GFcNHXQUZ+9kpnNq2ty2JBTRm2Ds19EdGgQGcPjiY8MYV9RNXuLqpk0OJZ/3zi7W3H0FdklNTy/6gA/P3ucJbw+TETWqGpGh/tZIvBPj366m/s+2kGTQ1l00hC+N28UafERR+1T19hEk0OJCDm+4Hi4vomfv7qB/27MIyo0iIYmBw8smsbCSSk8++U+/vDfrThUCQoQ/nTJZK44aQjgrCL6ywfbeDkzh+tPGcZvL5xIwDEPkvLDDZxx7yeEBQXwp8smc8a4gV26xmXbCvjRi+uoqG3khe+czCmjBnTp9xRX1XHFo1+xu7Ca2PBglv7sdBKjQgFYtbeEKx79ijEDo5g7JomM4fE0NDlYsaeEVXuLqalvYviASBqaHGTuL+WrO+eTGhvepTj6kpuezWTJ1nze//FcxqfEeDucLmtschAYIO1+Iemu6rpGyg83MCjOvXte29DEG+tyuWxGGiFB3a+5dzcR9Iu5hkzPyq+o5Z73tzF3TBJ/vGQSQxIi2twvNKj9b+PhIYE8fNV0JqRE88b6g/zl8ilMHxoPwLfmjGBMcjSPfLKbnywYw8xhCS3HJUWHcs/XpxAbHszjy/dSVdfEPV+fTFDgkX/0//x0NyXV9YxIjOSGp1dz2fTB/ObCdOIi3BuE3uRQHly6k4eW7iQ9NYbi6jr++uF2Xrn5lJb/8A6HsqOgkrEDo49LRK2VH27gm0+uIrfsMH++bDK/eTOLu9/bxr3fmEptQxO3v7aRtPhw3vj+HCJDj/x3unja0b2itx+q5JwHPuPT7YVcOWvoUbFW1jac8Noamhys3V/KrBEJHntgdUbmvhKWbM0HIK+8tt8mgjfW5XLH6xsJDgxg2IAIhg+I5LZzxjN0QNv/H7riF69uYMWeElbceaZbD/ZX1uTwP29ksbeomjvPm9BjcXTE5hryQ6+uycGhcNdFE9tNAu4QEX4wfwxLfnp6SxJoNmd0Iv+5afZRSaD1cb88bwI/XTCW19bm8D9vZtFcMs2vqOXpL/Zy8bRBvP/jufxw/mje2nCQxf9eg7ul18eX7+GhpTv5+ow0Xr/la/xw/hgy95fy6Y4j3ZDv+WAbCx9YzoUPf86ybQUtv7ukup7Pdxbx/MoD3PP+NhY9+hU7Cyp59JsZXDVrKN+ZO5JX1+Swam8JDyzZyd6iav582eSjkkBbxiZHkRobxifbj+4K/fDHu5h7zzLKaurbPE5Vuf21jSx6bAUfbsl36/o9SVW5+71tRLmu91B5250D+rpnvtjLj19az6RBsVw6fTCJUaG8n3WIZ77c12Pn2JlfybubDlFSXc/yne51gX8/Kw+Ax5bv4cvdRT0WS0csEfgZh0N5OTOb2SMSGJEY6bU4RIRbzxzDLfNG8cKqbJ7+Yh8ADyzZSZND+dmCcYQGBfKzs8dx10UTWbW3hA82u/cgXL6zkPTUGO79xhTCggO5ImMIafHh/PXDHagqr6/N4dFP93Dm+IFU1DZwwzOrOffB5cy5+2Nm/OEjrn1yJb/8v008/tkeahua+PvVMzh9rLML8w/nj2FwXDg/fXk9jy/fwxUZacwd03H3ZhHh9LFJfLGriIYmZztCk0N5ftV+KusaeX1tm72oeWDJTl5fm0uAwFsbDrp1/R3pTnXw0q0FZO4v5baF4xCBvLLDPRJTV+0qqOLl1dluX5Oqcv9HO7jr7S2cnZ7Mf26aze8vnsQzN8xi3rgk3svKw+Fm77byww3U1B/fe67ZPz7ZTXhwILHhwbztxr0rqa5nxZ4Svj1nBCMGRPKzlzdQXtPgVizdZYnAz6zcW8L+4hoWuersve3nZ4/j7PRk/vjfLTz9xV5ezszm6llDjyqeX3nSEMYMjOLu97ZS3+g4wW9zPlzXHyhj5rD4lmqUkKAAfnTmGDbllnPvh9u54/VNnDwygX9+cyYf/2wef7p0EhEhgcwYFs8vzxvPczfN5ss75rP9j+fyyS/O4OyJKS2/PzwkkLsumkhO6WEGRIbwq/PT3b7WeeOSqKxrZO3+UgA+21FIfkUdkSGBvLDqwHEPs1fX5PDg0p1cPjONq2cPZenWfKrb6LbrruKqOi56+HN+/86WLh3f5FD+94NtjEiM5KpZQxkYHUpeL5UI1meXtfkN+Q/vbOG21zby/KoDbv2eL3cXt/yd/uOaGYQFH6n+PG9yKnnltazPKTvh71BVXlp9gDl3f8z1T61qMwkdKK7hrQ0HuWb2UM6bnMJHW/I5XN/Uxm874qMth2hyKJfNGMwDV06jsLKOX72xqVuJ212WCPzMS6sPEB0WxLnd7EXTUwIChPsXTWNcSgy/e3sLoUEB/GD+mKP2CQoM4FfnT2BfcQ3/XnHi6dV3FlRSXd/EjGFxR22/dPpgRiZG8vdlu0mOCeUf18wkODCA4MAArpk9jNdvmcPfrprO4tNGMWd0IoPiwtvtDbMgPZnfXpjOY9dlEBse7Pa1zhmdSFCA8Imriuql1dkMiAzhjvMmsLOgirUHSlv2Xb2vhDte28ic0QP4f5dO5sIpg6htcLB0W4Hb52utvKaBa59cxcaccl5dk9NSKmlz38MN5JUf/03/uZX72ZFfxc/PHkdwYAApseG9lgjueG0jtzy3lrrGIw/T0up6vthVRFhwAHe9tZl1rf7+2vPJ9gJCAgP4w8WTjmqXAjgrPZngQOHdjXnHHdfkUMoPN7CroJJvPb2a21/bRFJ0KKv3lbZZUvvnZ7sJFOE7p43kwimDqK5vYtn2E9+797IOMSQhnImDYpiSFsdPFozlnY15vJ91qMPr6i5LBH6kvKaB97IOcfG0Qd3ultmTIkODeOL6DIYPiOCnC8aSFB163D7zxg1k7phEHlq6s936dIB1B5zf5qYPObrNIigwgF9fMIGRiZE8cd1JJER2b/bzG+aMYNqQuI53bCU6LJiZw+L5ZHshRVV1LNmaz2UzBnPZ9MFEhQbx/MpsACpqG/jxi+sZHB/OI9fOJCQogJOGJ5AcE+pWFcOxquoauf7pVewuqOJbXxtOZW0jq/aWHLffgeIa7nprM6f8eSnz7/30qH0y95Xwh3e2MHdMIudNdpaQBsWGtZkwelpu2WG2HaqkrKaBJVuOPEw/2HyIRofyxHUnkRwTxi3PraW4qu6Ev2v5ziJmDotv899/TFgwc8ck8V7WoZZv4ZW1DVzwt+WM+uW7TP3dh5x132es2lvC7y6ayEc/OY3Jg2P587vbjqoiOlRey6uZOXwjI43kmDBmjxxAYtSJ71354Qa+2FXEuZNSW0qyN58+iv+5IJ0zxnet11xnWCLwI29uyKWu0cGVJw3teOdeNjgunGU/n8dNc0e2u8+vzp9AZW0DP3pxPV/uLmpzpPLa/aUkRIYwrI2eH/PHJ/Pxz+cxLsV7g9nmjRvI1rwKHv10N40O5YqMIUSGBnHxtEG8s/Eg5TUN3PXWZvLKD3PfFdOICXOWOAIChPMnD+LT7YWUHz6+3ri+0cHXH/mSX7yygdxW9fZf7S7mqsdWsCm3nIevns5tC8cRGhTAR8c0PP/z093Mu3cZz63cz8JJKQyKC+NbT68ic18JeeWHufk/axkcF87DV81oeVClxIaRV17r8aqLZa5SUHRoEC9lZrdsf2djHsMHRDBn9AD+ee1MiqvrueW5te3WqxdW1rHtUCWnjkls91znTU4lt+wwG3LKAfjze9vYcrCCW+aN4tfnT+Duyybz4U9O4/qvDScoMIC7LkrnUEUt/1i2G4Cymnp+/UYWTarcfPooAAIDhAumpPLxtgIqa9uObenWfBqalIWTjlRDBgYIN5464qjqK0+xROAnVJUXV2WTnhrDpMGx3g6nTR11jRyfEsMvzhnP6n0lXP34Sk7589LjqorWZZcxfUhcn+hm2ZZ545wNy098vpfpQ+MY4xphfdWsodQ1OvjRS+t4fW0uPzhjNDOHHV2quXBqKvVNjuMe4gBb8ipYs7+UV9bkcMZfPuE3b2ax6NGvuOrxFeRX1PL3q6dz9sQUIkKCOHV0Iku25rc8wMtrGnho6U5OHZPE8tvmc98V03jhOyeTEhPGt55ezbeeWs3h+kYevy6D2IgjVWGDYsOpqW+iorbr7RbuWLatgCEJ4dwwZzjLdxZysOwwRVV1fLm7iAumDEJEmDQ4lnu+Ppk1+0s554HP+Hzn8e0JzW0Mp45uPxEsmOCqHtqU19J77Ka5I7lt4XhumjuSK2cNPaqn3cxhCVwybRCPLd/DU5/v5cy/fsqy7QXcds64o/a7YEoqdY2Olm63x3ov6xApMWFMS+tcKbOnWCLwE5/sKGRLXgXXnTLM26F0y/fmjWLNrxfw96tnMDg+nD+8vaVl+gtnHW4V04d65z+TO8anRJMSE4YqLMo40mA/aXAsU9Ji+WR7IVPTYvnhmWOOO3bakDjS4sPbrGLYkO2sEnvte6dw8bRB/GfFfvYVV3PXhel8dtsZR42sXpCeTE6ps7oF4PlVB6ipb+KOheNJiQ0DYGBMGM9/52QGRIWwPb+SB66c3pK0mjXv68nqodqGJr7YXcT8cQP5RsYQVOG1NTm8l3UIh8IFU49c16XTnd2FI0MDufbJlfzhnS1HlVY+31lEbHjwCb8IxUYEM2d0Iu9sOMjtr21kZGIkP10w9oQx3n7ueAJF+P07W0hLiODtH5zKd12lgWYzhsYzKDaMtzcc3/5QVdfIpzsKWTgp5YRjWjzJBpT5geYuc2nx4Xx9Zpq3w+m28JBAzp+SypjkKM6+/zNeXZPN4tNGsd71MDx2TENfIiLMnzCQt9Yf5IJjpqq4+fRR/P7tLdy3aBrBgcd/RxMRLpgyiMeX76Gkuv6odo4N2WUMjA5lxtB4Zg5L4M7zJhAZGtjmoMD5E5x1zku25DMqKYpnvtzLqaMTSR909MCwlNgwXv/e19hXXN3meJBBcc2JwHODyr7aU0xtg4Mzxg9kSEIEXxs1gFfW5JASG8aopEjGHZOcpqTF8d9b5/L7d7bw5Od7mTksnvMmp6KqfL6riDmjB3Q4JcZ5k1P5ZHshIvDqzad0WDWTGhvOQ1dNp7iqjm9kDGnz9wcECBdOG8QTy/eyv7iaYQOOdN1+d1Me9Y0Ozm1VLdTbrETgBz7eVsDGnHJunT+mzQdMfzU2OZpZwxN4fuUBHA5l3YFSRGBqJxtxe9ud547nnR+e2jIoq9l5k1P56s75jEqKavfYC6ak0uTQ46oY1meXMbVVlVhCZEi7I8MHRocxbUgcS7bm899NB8mvqOPGuSPa3HdAVGibSQAgxTVVhicHlS3bVkB4cCAnj3ROD3JFxhAOlNSwam9JS7XQscKCA/nDxZMYMzCKez/YTmOTgz1F1eSV1zLnBNVCzc5OTyYyJJDFc0e2e+3HWpCezJWzhp4wydw4ZwQhgQH89cMdLdsO1zdx/0c7mDQ4hpOGu3cuT/Cdp4Jpk6py/5IdDE2I4NIZvrcY3DUnD2VfcQ1f7i5m3YEyxiVHH/eA7Wuiw4IZ3s5gvo7aNiYOiiEpOpTPWo2SLq9pYE9Rdad6MS1IT2ZDTjkPLNnJmIFRzBvb+TU/BkaHEuDBQWWqysfbCpgzekDLt/KFk5wz4oKzzaQ9gQHCL84Zx56ial5dk9PSZjB3dMfXGRcRwpd3nskd547vgas4YmBMGN8+dThvbThIVq6zMfrJz/eQV17Lr89P91q1EFgi8HkfbcknK7eCW8/0rdJAs4WTUkiIDOHfK/axPrusT7cP9ITmEcrLdx7pNbUx11kl1tlEALC/uIab5o7oUuN6cGAASR4cVLaroIqc0sNHdZ8MCw7khjkjOG1sEqMHnrj314L0ZGYMjeOBJTtZsjWfIQnhbs8jFBse7JEOB989fRRxEcH85YPtFFTW8o9PdnPOxOSWEo+3+N6TwbRQVR5YspMRiZFcMm1Qxwf0Q6FBgXxjZhofbM6n/HBDn24f6Cmnj02i/HADG1wjYNcfKEMEJqe53xtszMAohiZEkBgVctwEeZ3hyUFlH7u6jR47++xPF4zlX9+e1eHxIsLtC8dzqKKW5TuLONWN0oCnxYQFc8u8UXy6o5DF/1pDfaODO87tvcnl2mOJwIdVHG5kS14Fi04actwoSl9yVavZPGf4eIkAnN0fAwQ+dU1gtyGnjFFJUS1jDtwhIty/aCqPXDuzW/3UPTmo7NMdhYxPiXZ7Cue2zB45oKXL7om6jfam604ZTmpsGOuzy7julOFenfOrme8+HQz5lc5vaoO78R+pPxieGMncMYnERQQzMrH9hlZfER8ZwtQhcXy6oxBVZX12OVO70P985rCEbjdQempQmcOhbMwpZ9aI7jeg/s8F6VwwJZXTx3m/RADO6q3fXjiRaUPiuPXM0d4OB/Bg91ERCQM+A0Jd53lVVX8rIs8ApwPlrl2/parrPRWHP8t3LTuZHBPm5Ug8795vTKWwss6rDW696fSxSTy4dCebD1ZQVFXHNC+VhFoPKuvMvEsd2V9SQ1VdI5MGdX/w46ikKB6+ekYPRNVzFk5KOWoUsbd5skRQB8xX1anANGChiJzs+uwXqjrN9WNJwEPyK5zzriTHHD93j69JjgnrsyOmPeH0sUmowt8+3gngtRGpnhpU1tyrZuLg/rnoTX/jsUSgTlWut8Gun76/LqYPaS4RDIz2/RKBv5mSFkdcRDAfbM4nJCjAa/MntR5UBlBT38j9H+2gop05ddyVdbCckMAAxnTQM8j0DI+2EYhIoIisBwqAj1R1peujP4nIRhG5X0R8/+uqlxRU1BIT1v0F4E3fExggLQviTBoU0yPr23ZF86CyvDJnInhpdTYPLt3Jv7q50tfm3ArGpUR77br8jUf/llW1SVWnAWnALBGZBNwJjAdOAhKA29s6VkQWi0imiGQWFrq3zJs5Wn5FnV+0D/ir5lXTvDmSunlQ2aHyw6gqz610LhDz/MoDNJ5gzYMTUVWyDpYzyaqFek2vpFtVLQOWAQtVNc9VbVQHPA202SFYVR9T1QxVzUhK6hut/f1NQWWtJQIfNm9cEknRoZw5PtlrMbQeVLZ6Xym7CqpYkJ7MwfLalnEAnZVbdpiymgYm9kBDsXGPxxKBiCSJSJzrdTiwANgmIqmubQJcAmR5KgZ/l19Rx0A/aCj2V4lRoaz+1VknnF+/N6S6BpU9v3I/0WFB3HfFVAbFhnW4mlx7snIrAOd0GqZ3eLJEkAosE5GNwGqcbQTvAM+JyCZgE5AI/NGDMfgtVbUSgekVqbFh7Miv5N1Nh7hs+mCiw4K5evZQlu8sYk9hVce/4BibD5YTGCBMSLVE0Fs82Wtoo6pOV9UpqjpJVX/v2j5fVSe7tl3bqmeR6UGlNQ00NCkD21j20ZielBobTkFlHfVNDq6e7Vzv4oqThhAcKC1tBp2RlVvO6KSoXlmZyzhZk7yP8qfBZMa7Ul1jCTKGxbd0Yx0YHcbCSam8kpnN4fqmEx1+nKyDFTZ+oJdZIvBRRxKBlQiMZzXPBXT17KPXwr5m9lAqahvbXZ6xLQUVtRRW1vXIiGLjPksEPqrANarYBpMZTztzwkD+fNlkLjpmxbWZw+IJCQwg62B5O0cer3lffxol3hf07RU8TJe1jCq2EoHxsLDgwKNmgG0WHBjAmOQothyscPt3NfcYOnbZTONZViLwUQWVdcRHBLe7XKExvWFCagxb89xPBBtzyhiZGNnnV5nzNZYIfFR+hXUdNd6XnhpDUVU9BZXtL15TUdvAv1fs5/yHlrNkawGzR3pv7V5/ZWnXR+VX1jHQEoHxsuYqni0HKxg47vh/j6rK5Y98yY78KiakxvD7iydy+cy03g7T71ki8FEFFbWMGej7i7SYvm1CiisR5FUw75glJ8E5a+mO/Cp+cc44bpk3yiPrBJuOWdWQD3I4lILKOus6arwuNiKYwXHhbM2rbPPzTa51B04ZNcCSgBdZIvBBxdX1NDnU2ghMnzAhNYYt7XQhzcotJ0COlByMd1gi8EG2II3pS9IHxbC3qLrNEcZZueWMGRhta2Z4mSUCH9TcQ8OqhkxfkJ4ag0Nhe/7R1UOqyqbcChs81gdYIvBBBS1rFVuJwHhfeuqRnkOt5VfUUVRVx2SbV8jrLBH4oOZF65Ns5lHTB6TFhxMdGnTcwLLmhuLJaVYi8DZLBD4ov7KWxKgQggPt9hrvCwgQxqdGs6WNRBAg2LoDfYA9KXxQQUWtNRSbPiXdNdWEw6Et2zbnljMqKYqIEBvO5G2WCHyQLVFp+pr0QTHU1DdxoKSmZdum3HImW0Nxn+DJNYvDRGSViGwQkc0i8jvX9hEislJEdonISyIS4qkY/FV+RS3JViIwfUhz9c9GV7tAQUUtBZV11mOoj/BkiaAOmK+qU4FpwEIRORm4B7hfVUcDpcCNHozB7zQ2OSiqslHFpm8ZlxLN4Lhw/vLBNsoPN1hDcR/jyTWLtdV6xMGuHwXmA6+6tj8LXOKpGPxRSXU9DoUk6zpq+pDQoED+dvV08spque3VDWzMKUfkSNdS410ebSMQkUARWQ8UAB8Bu4EyVW107ZIDDG7n2MUikikimYWFhZ4M06cUV9cDkBhpNW6mb5kxNJ47zh3PB5vzeeqLvYxMjCTS1h3oEzyaCFS1SVWnAWnALGB8J459TFUzVDUjKSnJYzH6mlJXIoi3RGD6oBtPHcFZE5KprG20huI+pFd6DalqGbAMOAWIE5HmrwFpQG5vxOAvmksEAywRmD5IRPjrN6Zy0vB4Fk5K9XY4xsWTvYaSRCTO9TocWABsxZkQLnftdj3wpqdi8EelNVYiMH1bbEQwr9z8NRZOSvF2KMbFkxV0qcCzIhKIM+G8rKrviMgW4EUR+SOwDnjSgzH4neIqZyKICw/2ciTGmP7CY4lAVTcC09vYvgdne4HxgNKaeuIiggmy6SWMMW6yp4WPKa6uJyHCqoWMMe6zROBjSqvrSbD2AWNMJ1gi8DEl1fXWUGyM6RRLBD6mpLreuo4aYzrFEoEPUVVKa6xEYIzpHEsEPqSyrpGGJrUSgTGmUywR+JCW6SWs15AxphMsEfiQ5uklEqIsERhj3GeJwIc0lwhsHIExpjMsEfiQlhKBtREYYzrBEoEPKbVEYIzpAksEPqSkup7QoAAiQgK9HYoxph+xROBDSlzTS4iIt0MxxvQjlgh8SInNM2SM6QJLBD6kpMYSgTGm8ywR+JCS6nobTGaM6TRPLlU5RESWicgWEdksIj9ybb9LRHJFZL3r5zxPxeBvrGrIGNMVnlyqshH4maquFZFoYI2IfOT67H5VvdeD5/Y79Y0OKmsbLREYYzrNk0tV5gF5rteVIrIVGOyp8/m7shobQ2CM6ZpeaSMQkeE41y9e6dr0AxHZKCJPiUh8b8Tg62xUsTGmqzyeCEQkCngN+LGqVgCPAKOAaThLDH9t57jFIpIpIpmFhYWeDrPfs1HFxpiu8mgiEJFgnEngOVV9HUBV81W1SVUdwOPArLaOVdXHVDVDVTOSkpI8GaZPsBKBMaarPNlrSIAnga2qel+r7amtdrsUyPJUDP6k1NoIjDFd5MleQ3OAbwKbRGS9a9svgatEZBqgwD7gux6MwW+UuEoEceHBXo7EGNPfeLLX0OdAW5PevOupc/qzkup64iKCCQq0MYLGmM6xp4aPKKmutwVpjDFdYonAR9ioYmNMV1ki8BEl1fXEWyIwxnSBJQIfUVJdzwBLBMaYLrBE4ANUldIaKxEYY7rGEoEPqKxrpKFJrURgjOkSSwQ+oHl6CVuLwBjTFZYIfEBhZR0AA6IsERhjOs/tRCAip4rIDa7XSSIywnNhmc4ocCWCgdFhXo7EGNMfuZUIROS3wO3Ana5NwcB/PBWU6ZyCiloABsaEejkSY0x/5G6J4FLgIqAaQFUPAtGeCsp0TkFlHUEBYiOLjTE54TgnAAAUMklEQVRd4m4iqFdVxTlRHCIS6bmQTGcVVNaRGBVKQEBbUzsZY8yJuZsIXhaRR4E4EfkOsATnWgKmDyiorLNqIWNMl7k1+6iq3isiC4AKYBzwG1X9qIPDTC8pqKglLT7c22EYY/qpDhOBiAQCS1T1DMAe/n1QYWUd04fa0s/GmK7psGpIVZsAh4jE9kI8ppMamhwUV9czMNqqhowxXePuwjRVOFca+whXzyEAVb3VI1EZtxVVucYQWBuBMaaL3E0Er7t+3CYiQ4B/Ack4exs9pqoPikgC8BIwHOdSlVeoamlnfrc5oqDCBpMZY7rH3cbiZ0UkBBjr2rRdVRs6OKwR+JmqrhWRaGCNq0TxLWCpqt4tIncAd+AcrGa64MioYisRGGO6xt2RxfOAncDfgX8AO0TktBMdo6p5qrrW9boS2AoMBi4GnnXt9ixwSZciNwAUVNqoYmNM97hbNfRX4GxV3Q4gImOBF4CZ7hwsIsOB6cBKIFlV81wfHcJZddTWMYuBxQBDhw51M0z/U1BRhwgkRlkiMMZ0jbsDyoKbkwCAqu7AOd9Qh0QkCngN+LGqVrT+rPVo5WOp6mOqmqGqGUlJSW6G6X8KKutIiAghONAmkjXGdI27JYJMEXmCIxPNXQNkdnSQiATjTALPqWpzY3O+iKSqap6IpAIFnQ3aHFFYWUuStQ8YY7rB3a+R3wO2ALe6fra4trVLRAR4Etiqqve1+ugt4HrX6+uBNzsTsDlaYWUdA2Osx5AxpuvcLREEAQ82P9Bdo407+ho6B/gmzvEH613bfgncjXPuohuB/cAVnY7atCiorGNMsk0Ea4zpOncTwVLgLJwDywDCgQ+Br7V3gKp+DrQ3HeaZ7gZo2udwqLNEYFVDxphucLdqKExVm5MArtcRngnJuKu0pp5Gh1oiMMZ0i7uJoFpEZjS/EZEM4LBnQjLuahlMZm0ExphucLdq6MfAKyJy0PU+FVjkmZCMu2xUsTGmJ5ywRCAiJ4lIiqquBsbjnCOoAXgf2NsL8ZkTaFmr2OYZMsZ0Q0dVQ48C9a7Xp+Ds9fN3oBR4zINxGTccqRqyEoExpus6qhoKVNUS1+tFOGcQfQ14rVWXUOMlhZV1RIcFERYc6O1QjDH9WEclgkARaU4WZwIft/rM3fYF4yEFlbXWPmCM6baOHuYvAJ+KSBHOXkLLAURkNFDu4dhMBwoq6qx9wBjTbSdMBKr6JxFZirOX0IeuSeLAWZL4oaeDMydWUFnH9KFx3g7DGNPPdVi9o6or2ti2wzPhGHepqlUNGWN6hM1d3E9V1jVS2+CwqiFjTLdZIuinmtcqtimojTHdZYmgn8oprQEgJdZKBMaY7rFE0E+tzy5DBCYOivF2KMaYfs4SQT+19kAZ45KjiQ5za8VQY4xplyWCfsjhUNYfKGX60Hhvh2KM8QEeSwQi8pSIFIhIVqttd4lIroisd/2c56nz+7I9RVVU1DbaGAJjTI/wZIngGWBhG9vvV9Vprp93PXh+n7V2fxkAM6xEYIzpAR5LBKr6GVDS4Y6m09ZllxITFsTIxEhvh2KM8QHeaCP4gYhsdFUd2VfaLli7v4zpQ+MJCGhvSWhjjHFfbyeCR4BRwDQgD/hrezuKyGIRyRSRzMLCwt6Kr8+rqG1gR0GlVQsZY3pMryYCVc1X1SZVdQCPA7NOsO9jqpqhqhlJSUm9F2QftzG7HFWsodgY02N6NRGISGqrt5cCWe3ta9q29kApIjDNEoExpod4bHEZEXkBmAckikgO8FtgnohMAxTYB3zXU+f3VesOlDI6KYoYG0hmjOkhHksEqnpVG5uf9NT5/IGqsi67jHPSU7wdijHGh9jI4n5kT1E1ZTUNzBhm1ULGmJ5jiaAfWbOvFLCBZMaYnmWJoB/5ak8xiVEhjB4Y5e1QjDE+xBJBP6GqfLW7mNkjByBiA8mMMT3HEkE/sa+4hkMVtZwycoC3QzHG+BhLBP3EV7uLATjZEoExpodZIugnvtpTTFJ0KKOSbKI5Y0zPskTQD6gqK/YUc4q1DxhjPMASQT+wu7Cawso6Thll1ULGmJ5niaAf+GqPs33AGoqNMZ7gsSkmTOepKjmlh1m1t4SahiYWZQwhJCiAFbuLSY0NY9iACG+HaIzxQZYI+oilW/P59RtZ5JXXtmx7eXU2D101nRV7ijl9bJK1DxhjPMISQR/x6poc6hsd/O6iicwakcD+4hpuf20jCx/4jLpGBydb+4AxxkMsEfQRG7LL+NroRK7/2nAAJqTGMDktlh+/uI4N2eXMGZ3o3QCNMT7LEkEfUFBZy8HyWr6dFnvU9sFx4by4+BSKq+oYGBPmpeiMMb7Oeg31ARuzywGYOuT46aUDA8SSgDHGoywR9AEbcsoIDBAmDorxdijGGD/ksUQgIk+JSIGIZLXaliAiH4nITtefNrE+sD67jLHJ0USEWE2dMab3ebJE8Ayw8JhtdwBLVXUMsNT13q+pKhtzypl6TPuAMcb0Fo8lAlX9DCg5ZvPFwLOu188Cl3jq/P3F/uIayg83tNk+YIwxvaG32wiSVTXP9foQkNzL5+9zNuSUATA1zRKBMcY7vNZYrKoKaHufi8hiEckUkczCwsJejKx3bcguJyw4gLHJtvykMcY7ejsR5ItIKoDrz4L2dlTVx1Q1Q1UzkpKSei3A3rYhp4xJg2IJCrQOXMYY7+jtp89bwPWu19cDb/by+fuUhiYHWbnl1j5gjPEqT3YffQH4ChgnIjkiciNwN7BARHYCZ7ne+63thyqpa3RYIjDGeJXHOq6r6lXtfHSmp87Z32zMcY0otq6jxhgvsoppL/p8VyHxEcEMTbB1Bowx3mOJwEt2FVTxXtYhrjhpiK0zYIzxKksEXvL3ZbsICwrkO3NHejsUY4yfs0TgBXsKq3hzfS7XnjyUxKhQb4djjPFzlgi84O/LdhMcGMDi00Z5OxRjjLFE0Nv2F1fzxvpcrpk9jKRoKw0YY7zPEkEvKqqq447XNhEYINx8urUNGGP6BpsAv5cs2ZLPHa9vpOJwI7+/eKKtOmaM6TMsEfSC+z/awYNLdzIhNYbnbprGuJRob4dkjDEtLBF4WH5FLf/4ZBfnT07lvkVTCQ0K9HZIxhhzFGsj8LBnvtxHo0O5beE4SwLGmD7JEoEHVdc18tyK/SycmMKwAZHeDscYY9pkicCDXsnMpqK2kZts9LAxpg+zROAhTQ7lyS/2MmNoHDOHxXs7HGOMaZclAg/5YPMhsksOs/g0Kw0YY/o26zXUg4qq6vh0eyFf7Snm420FDBsQwYL0FG+HZYwxJ2SJoIdU1jZw7oPLKaysIz4imNkjBvDd00cSGGBTTBtj+javJAIR2QdUAk1Ao6pmeCOOnvTU5/sorKzjmRtO4rQxSQRYAjDG9BPeLBGcoapFXjx/jymrqeeJ5Xs4Oz2ZeeMGejscY4zpFGss7gH//HQPVfWN/Ozscd4OxRhjOs1biUCBD0VkjYgsbmsHEVksIpkikllYWNjL4bmvoLKWZ77cy0VTB9kcQsaYfslbieBUVZ0BnAt8X0ROO3YHVX1MVTNUNSMpKan3I3TTP5btpqFJ+clZY70dijHGdIlXEoGq5rr+LAD+D5jljTi6q7CyjudXHuDyGWkMT7QpJIwx/VOvJwIRiRSR6ObXwNlAVm/H0ROeW7mf+iYH37VFZowx/Zg3eg0lA/8nIs3nf15V3/dCHN1S19jEf1bsZ/74gYxMivJ2OMYY02W9nghUdQ8wtbfP2x0Oh/LG+lxOG5tEYpRzneG3N+RRVFXPt+eM8HJ0xhjTPdZ91A2f7Cjgpy9v4NonVlJe04Cq8tTnexmbHMWc0QO8HZ4xxnSLJQI3vJKZQ3RoEHsKq7nhmVV8sr2QLXkV3DBnBK4qLmOM6bcsEXSguKqOJVvzueKkITx01TTWZ5ex+N+ZxEcEc+n0wd4Ozxhjus0SQQfeWH+QhibliowhLJyUyt1fn0JDk3LtycMIC7alJ40x/Z/NPnoCqsormdlMTYttGTV8RcYQMobF29KTxhifYSWCE8jKrWDboUq+kTHkqO0jk6JsemljjM+wRHACL2dmExoUwIVTB3k7FGOM8RirGmpDXWMTG3PKeXN9LgsnpRAbHuztkIwxxmMsEbjsK6rmwy2HWLq1gHXZZdQ3OggJDOC6U4Z7OzRjjPEov08Eaw+U8svXN7HtUCUAE1JjuO7kYZw0IoGThieQEBni5QiNMcaz/DoRbMgu4/onVxEXGcxvLkhnQXoyQxIivB2WMcb0Kr9NBFm55XzzyZXER4bw0ndPJjU23NshGWOMV/hlr6HNB51JIDosmOe/M9uSgDHGr/l0IqhtaEJVj9r25e4irnx0BWHBgTz/ndmkxVtVkDHGv/l01dDDH+/i7Y0HuXjaYC6ZNogteRX89KUNDE+M4JkbZjEozkoCxhjj04lg0uBY1mWX8rePd/LQ0p0AzBqewOPXZRAbYWMDjDEGvJQIRGQh8CAQCDyhqnd74jwLJ6WwcFIK+RW1vL3hIFV1jdx8+iibLM4YY1rp9UQgIoHA34EFQA6wWkTeUtUtnjpnckwYN821dYWNMaYt3mgsngXsUtU9qloPvAhc7IU4jDHG4J1EMBjIbvU+x7XNGGOMF/TZ7qMislhEMkUks7Cw0NvhGGOMz/JGIsgFWk/wn+badhRVfUxVM1Q1IykpqdeCM8YYf+ONRLAaGCMiI0QkBLgSeMsLcRhjjMELvYZUtVFEfgB8gLP76FOqurm34zDGGOPklXEEqvou8K43zm2MMeZofbax2BhjTO+QYydl64tEpBDY34lDEoEiD4XTl/njdfvjNYN/Xrc/XjN077qHqWqHvW36RSLoLBHJVNUMb8fR2/zxuv3xmsE/r9sfrxl657qtasgYY/ycJQJjjPFzvpoIHvN2AF7ij9ftj9cM/nnd/njN0AvX7ZNtBMYYY9znqyUCY4wxbvK5RCAiC0Vku4jsEpE7vB2PJ4jIEBFZJiJbRGSziPzItT1BRD4SkZ2uP+O9HWtPE5FAEVknIu+43o8QkZWu+/2Sa9oSnyIicSLyqohsE5GtInKKr99rEfmJ6992loi8ICJhvnivReQpESkQkaxW29q8t+L0kOv6N4rIjJ6Kw6cSQatFb84F0oGrRCTdu1F5RCPwM1VNB04Gvu+6zjuApao6Bljqeu9rfgRsbfX+HuB+VR0NlAI3eiUqz3oQeF9VxwNTcV6/z95rERkM3ApkqOoknFPRXIlv3utngIXHbGvv3p4LjHH9LAYe6akgfCoR4CeL3qhqnqqudb2uxPlgGIzzWp917fYscIl3IvQMEUkDzgeecL0XYD7wqmsXX7zmWOA04EkAVa1X1TJ8/F7jnP4mXESCgAggDx+816r6GVByzOb27u3FwL/UaQUQJyKpPRGHryUCv1v0RkSGA9OBlUCyqua5PjoEJHspLE95ALgNcLjeDwDKVLXR9d4X7/cIoBB42lUl9oSIROLD91pVc4F7gQM4E0A5sAbfv9fN2ru3Hnu++Voi8CsiEgW8BvxYVStaf6bO7mA+0yVMRC4AClR1jbdj6WVBwAzgEVWdDlRzTDWQD97reJzffkcAg4BIjq8+8Qu9dW99LRG4teiNLxCRYJxJ4DlVfd21Ob+5qOj6s8Bb8XnAHOAiEdmHs8pvPs668zhX9QH45v3OAXJUdaXr/as4E4Mv3+uzgL2qWqiqDcDrOO+/r9/rZu3dW48933wtEfjFojeuuvEnga2qel+rj94Crne9vh54s7dj8xRVvVNV01R1OM77+rGqXgMsAy537eZT1wygqoeAbBEZ59p0JrAFH77XOKuEThaRCNe/9eZr9ul73Up79/Yt4DpX76GTgfJWVUjdo6o+9QOcB+wAdgO/8nY8HrrGU3EWFzcC610/5+GsM18K7ASWAAnejtVD1z8PeMf1eiSwCtgFvAKEejs+D1zvNCDTdb/fAOJ9/V4DvwO2AVnAv4FQX7zXwAs420EacJb+bmzv3gKCs1fkbmATzl5VPRKHjSw2xhg/52tVQ8YYYzrJEoExxvg5SwTGGOPnLBEYY4yfs0RgjDF+zhKB8Wki0iQi61v9nHByNhG5WUSu64Hz7hORxC4cd46I/M41A+V73Y3DGHcEdbyLMf3aYVWd5u7OqvpPTwbjhrk4B07NBT73cizGT1iJwPgl1zf2/xWRTSKySkRGu7bfJSI/d72+1bXmw0YRedG1LUFE3nBtWyEiU1zbB4jIh6459J/AOfin+VzXus6xXkQedU2Xfmw8i0RkPc7plx8AHgduEBGfGxlv+h5LBMbXhR9TNbSo1WflqjoZeBjnw/dYdwDTVXUKcLNr2++Ada5tvwT+5dr+W+BzVZ0I/B8wFEBEJgCLgDmukkkTcM2xJ1LVl3DOIpvlimmT69wXdefijXGHVQ0ZX3eiqqEXWv15fxufbwSeE5E3cE7tAM7pPb4OoKofu0oCMTjXDLjMtf2/IlLq2v9MYCaw2jltDuG0P0HcWGCP63WkOteaMMbjLBEYf6btvG52Ps4H/IXAr0RkchfOIcCzqnrnCXcSyQQSgSAR2QKkuqqKfqiqy7twXmPcZlVDxp8tavXnV60/EJEAYIiqLgNuB2KBKGA5rqodEZkHFKlzLYjPgKtd28/FOTEcOCcPu1xEBro+SxCRYccGoqoZwH9xzsP/vzgnTJxmScD0BisRGF8X7vpm3ex9VW3uQhovIhuBOuCqY44LBP7jWipSgIdUtUxE7gKech1Xw5Hpgn8HvCAim4EvcU6ljKpuEZFfAx+6kksD8H1gfxuxzsDZWHwLcF8bnxvjETb7qPFLrgVuMlS1yNuxGONtVjVkjDF+zkoExhjj56xEYIwxfs4SgTHG+DlLBMYY4+csERhjjJ+zRGCMMX7OEoExxvi5/w/zwtGwiyLHogAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot scores\n",
    "with open(\"scores_file.json\", \"r\") as read_file:\n",
    "    scores = json.load(read_file)\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1: 36.23999918997288\n",
      "Episode 2: 3.4799999222159386\n",
      "Episode 3: 31.049999305978417\n",
      "Mean score is:  23.58999947272241\n"
     ]
    }
   ],
   "source": [
    "#Inference\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "agent = Agent(state_size=env_info.vector_observations.shape[1], action_size=brain.vector_action_space_size, \n",
    "              num_agents=env_info.vector_observations.shape[0],  random_seed=0)\n",
    "#Since the model is trained on gpu, need to load all gpu tensors to cpu:\n",
    "agent.actor_local.load_state_dict(torch.load('checkpoint_actor.pth', map_location=lambda storage, loc: storage))\n",
    "\n",
    "agent.eps = 0.05\n",
    "scores_list = []\n",
    "def ddpg_inference(n_episodes=3):\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]     # reset the environment\n",
    "        states = env_info.vector_observations                   # get the current states (for all agents)\n",
    "        agent.reset() #reset the agent OU Noise\n",
    "        scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "        while True:\n",
    "            actions = agent.act(states)\n",
    "            env_info = env.step(actions)[brain_name]           # send all actions to the environment\n",
    "            next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "            rewards = env_info.rewards                         # get rewards (for each agent)\n",
    "            dones = env_info.local_done                        # see if episode finished\n",
    "            scores += rewards                                  # update the score (for each agent)\n",
    "            states = next_states                               # roll over states to next time step\n",
    "            if np.any(dones):                                  # exit loop if episode finished\n",
    "                break\n",
    "        print('Episode {}: {}'.format(i_episode, np.mean(scores)))\n",
    "        scores_list.append(np.mean(scores))\n",
    "    print('Mean score is: ', np.mean(np.array(scores_list)))\n",
    "\n",
    "ddpg_inference()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
